Objectives

• Introduce Bayesian Linear Regression (BLR): Understand its principles and how it differs from traditional methods.

• Explain Bayesian Concepts: Highlight Bayes’ Theorem, prior knowledge, and posterior distributions.

• Discuss Practical Applications: Show how BLR is applied in analyzing real-world data, like airline delays.

• Explore Advantages of Bayesian Methods: Quantifying uncertainty, improving predictions, and handling complex data.

• Present Analysis Findings: Summarize key insights from our BLR model on time and weather-related airline delays.

What is Bayesian Linear Regression?

• BLR: A statistical approach combining prior knowledge and new data.

• Goal: Model relationships, make predictions, and handle uncertainty in estimates.

• Difference from Traditional Methods: Probability-based estimates instead of fixed values.

Introduction to Bayesian Linear Regression

• Regression under the frequentist framework
  • Independent variables are used to predict dependent variables
  • Linear regression finds best-fitting line to observed data to make further predictions
    • Regression parameters (\beta) are assumed to be fixed
  • Only collected data is used for approximation
• Regression under the Bayesian framework
  • Independent variables are used to predict dependent variables
  • Regression parameters (\beta) are not assumed to be fixed
  • Collected data is used alongside prior knowledge for approximation

Why Bayesian?

Advantages of Bayesian Linear Regression[1]

• Incorporation of Prior Knowledge

• Uncertainty Quantification

• Expanded Hypotheses

• Automatic Meta-Analyses

• Improved Handling of Small Samples

• Complex Model Estimation

Steps in Bayesian Linear Regression

1. Model Specification: Define the linear relationship between the dependent and independent variables.

2. Choose Priors: Select prior distributions for the model parameters, reflecting any existing knowledge about their values.

3. Data Collection: Gather relevant data for the variables in the model.

4. Model Fitting: Use computational methods, such as Markov Chain Monte Carlo (MCMC), to estimate the posterior distributions of the parameters based on the observed data.

5. Result Interpretation: Analyze the posterior distributions to understand the relationships between variables, including estimating means and credible intervals.

Frequentist vs. Bayesian Approach

• The Frequentist Approach
  • Typical linear model

Y = \beta_0 + \beta_1X + \varepsilon

• Y : Dependent variable, the outcome
• \beta_0 : y intercept
• \beta_1 : The regression coefficient
• X : Independent variable
• \varepsilon : Random error [2]
• \hat\beta provides a point estimate for the regression coefficient in the frequentist framework

Frequentist vs. Bayesian Approach

• The Bayesian Approach
  • A regression is constructed using probability distributions, not point estimates as in the frequentist approach
  • Bayes Rule is used to inform the model [2]

p(B|A) = \frac{p(A|B)\cdot p(B)}{p(A)}

• Bayes Rule allows for the calculation of inverse probability (p(B|A) \text{ from } p(A|B))
• p(B|A) \text{ and } p(A|B) are conditional probabilities
• p(A) \text{ and } p(B) are marginal probabilities [3]

The Bayesian Approach

• A more formal expression of Bayes Rule applied for continuous parameters

\begin{align*} p(\theta|y) =& \frac{ L(\theta|y)p(\theta) }{p(y)}\\ \\ p(\theta|y) \propto & \text{ }L(\theta|y)p(\theta) \end{align*}

• The normalization constant (p(y) above) ensures the posterior distribution is a valid distribution
  • The posterior density function can be written without this constant
• The resulting prediction is not a point estimate, but a distribution [5]

The Bayesian Approach

• The Bayesian Linear Regression Model
  • Changes based on:
    • distribution chosen for regression
    • distribution and hyperparameters chosen for priors
  • Our test cases models a continuous outcome so we use a Normal model with conjugate normal priors

Role of Prior Knowledge in Shaping Predictions

• Priors can be subjective or objective
  • objective is preferred
• Uninformative priors can be used when there is not adequate prior knowledge
• Discounted priors are the result of adjusting a known prior to better reflect the current data[3]

Figure from [3]

Understanding the Bayesian Framework

Interpreting the Posterior with Markov Chain Monte Carlo (MCMC)

Sara’s Prior Selection & Model Specification

Model 1: Continuous Predictor

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \end{align*} Where:

• Y_i is the arrival delay for the i-th flight
• X_i is the departure delay for the i-th flight
• \mu_i = \beta_0 + \beta_1X_i is the local mean arrival delay, specific to the departure time
• \sigma^2 is the variance of the errors
• \overset{\text{ind}}{\sim} indicates conditional independence of each arrival delay with the given parameters

Model 1: Continuous Predictor

The model can be written as

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \\ \beta_{0} &\sim N(m_0, s_0^2)\\ \beta_1 &\sim N(m_1, s_1^2)\\ \sigma &\sim \text{Exp}(l) \end{align*}

• Regression parameters

  • Intercept: \beta_0 \sim N(m_0, s^2_0)
  • Slope: \beta_1 \sim N(m_1, s^2_1)
  • Error: \sigma \sim \text{Exp}(l)

Model 2: Categorical Predictor

\begin{align*} Y_i|\beta_0, \beta_1, ... \beta_6, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_{i1} + \beta_2X_{i2} + ... \beta_6X_{i6} \\ \beta_{0} &\sim N(m_0, s_0^2)\\ \beta_1 &\sim N(m_1, s_1^2)\\ \sigma &\sim \text{Exp}(l) \end{align*}

Where

• X_{i1}, X_{i2}, ..., X_{i6} are indicator variables for the day of the week.

Tuning Hyperparameters

• Mean, \mu = 2.12, and SD, \sigma = 36.42, of arrival delays were taken for departures within 10 minutes of the mean departure time.

\beta_{0c} \sim N(2, 36^2)

• The coefficient, 0.019, and SE of the OLS regression were used.

\beta_{1} \sim N(0.02, 0.01^2)

• The expected standard deviation was set equal to the residual standard error of the regression, i.e. E(\sigma) = \frac{1}{l} \approx 51.1

\sigma \sim \text{Exp}(0.02)

The Updated Models

Model 1

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \\ \beta_{0} &\sim N(2, 36^2)\\ \beta_1 &\sim N(0.02, 0.01^2)\\ \sigma &\sim \text{Exp}(0.02) \end{align*}

Model 2

\begin{align*} Y_i|\beta_0, \beta_1, ... \beta_6, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_{i1} + \beta_2X_{i2} + ... \beta_6X_{i6} \\ \beta_{0} &\sim N(2, 46^2)\\ \beta_j &\sim N(0, 50^2)\\ \sigma &\sim \text{Exp}(0.02) \end{align*}

Statistical Programming

• Data was analyzed with R [8] in RStudio [9], imported via CSV.
  • Data sourced from the Bureau of Transportation Statistics via Kaggle [10].
• Libraries
  • rstanarm [11]
    • stan_glm() - simulation of model
    • posterior_predict() - simulation of posterior
  • bayesrules [12]
    • prediction_summary_cv() - evaluation of posterior
  • bayesplot [13]
    • Effective sample size (neff()) and R-hat (rhat())

Heather’s Prior Selection & Model Specification

Prior Selection

• Intercept (\beta_0): \beta_0 \sim N(0, 5^2) Assumes no strong baseline effect.

• Slope (\beta_1): \beta_1 \sim N(0, 5^2) Reflects no strong prior belief about the relationship between weather incidents and delays.

• Error Term (\sigma): \sigma \sim \text{Exp}(1) Accounts for variability in delays; allows flexibility.

Model Specification

Y_i \mid \beta_0, \beta_1, \sigma \sim N(\mu_i, \sigma^2) \mu_i = \beta_0 + \beta_1 X_i

• Y_i: Arrival delay (minutes)
  
• X_i: Weather-related incidents

The Dataset

Table 1

Data Preprocessing

Comparison of the Models

Effective Sample Size

\hat{R} Analysis

OLS Comparison

Posterior Predictive Checks: Model 2

Exploring the Data

Exploring the Data

Choosing Focus

Code for Model

Trace Plots and Posterior Distributions

Model Parameters and Estimates

Posterior Distribution for Weather Count Coefficient

Posterior Predictive Check

Sara’s Conclusions

• Model 1: An increase of 1.2 seconds in arrival delay per minute past midnight

Y = -11.66 + 0.02X.

• Model 2: Varying predicted arrival delays for the days of the week

Y = 1.54 + 4.96X_1 + 2.70X_2 + 1.91X_3 + 3.50X_4 + 4.41X_5 + 2.99X_6.

• Residual Variability
  • \sigma \approx 51 for all models

Model Diagnostics

• K-Fold Cross Validation

  • MAE of 15.7 and 17.1 \text{ minutes}
    
  • 96.6\% of observations within 95\% prediction intervals

• Effective Sample Size

  • N_\text{eff} > 0.1

• \hat{R} \approx 1

• MCMC Diagnostics

  • Good mixing and convergence
  • Consistency and unimodality

• Autocorrelation

  • Weak correlation, rapid decay

Heather’s Conclusions

Key Findings

• Intercept: -2116.53 (95% CI: [-2131.41, -2100.91])

  • Indicates significantly shorter delays without weather incidents.

• Weather Count Coefficient: 1041.97 (95% CI: [1036.73, 1047.15])

  • A 1-unit increase in weather incidents leads to an average 1042-minute delay.

  • Weather incidents are infrequent but highly disruptive.

• Uncertainty Measures:

  • Residual variability: Standard deviation = 8676.19.

  • Suggests other unmeasured factors affecting delays.

• Model Diagnostics:

  • Rhat = 1.00 for all parameters, indicating convergence.

  • Large effective sample sizes ensure reliable posterior estimates.

Conclusion

• Key Insight:

  • Weather-related incidents, though infrequent, have a disproportionately large impact on delay times.

  • Highlights the need for better weather management and forecasting.

• Bayesian Approach:

  • Accounts for uncertainty, providing credible intervals for estimates.

  • Supports informed decision-making in airline operations and policy-making.

Discussion and Future Research

Thank You! Questions?

Prior Summary: Model 1 Default Priors

Priors for model 'default_model_dt' 
------
Intercept (after predictors centered)
  Specified prior:
    ~ normal(location = 4.5, scale = 2.5)
  Adjusted prior:
    ~ normal(location = 4.5, scale = 129)

Coefficients
  Specified prior:
    ~ normal(location = 0, scale = 2.5)
  Adjusted prior:
    ~ normal(location = 0, scale = 0.43)

Auxiliary (sigma)
  Specified prior:
    ~ exponential(rate = 1)
  Adjusted prior:
    ~ exponential(rate = 0.019)
------
See help('prior_summary.stanreg') for more details

Prior Summary: Model 1 Flat Priors

Priors for model 'flat_model_dt' 
------
Intercept (after predictors centered)
 ~ flat

Coefficients
 ~ flat

Auxiliary (sigma)
 ~ flat
------
See help('prior_summary.stanreg') for more details

Prior Summary: Model 2 Default Priors

Priors for model 'default_model_dow' 
------
Intercept (after predictors centered)
  Specified prior:
    ~ normal(location = 4.5, scale = 2.5)
  Adjusted prior:
    ~ normal(location = 4.5, scale = 129)

Coefficients
  Specified prior:
    ~ normal(location = [0,0,0,...], scale = [2.5,2.5,2.5,...])
  Adjusted prior:
    ~ normal(location = [0,0,0,...], scale = [364.46,361.83,370.01,...])

Auxiliary (sigma)
  Specified prior:
    ~ exponential(rate = 1)
  Adjusted prior:
    ~ exponential(rate = 0.019)
------
See help('prior_summary.stanreg') for more details